Methods for controlling use of computing resources, such as virtual game consoles

ABSTRACT

An artificial intelligent agent can act as a player in a video game, such as a racing video game. The game can be completely external to the agent and can run in real time. In this way, the training system is much more like a real world system. The consoles on which the game runs for training the agent are provided in a cloud computing environment. The agents and the trainers can run on other computing devices in the cloud, where the system can choose the trainers and agent compute based on proximity to console, for example. Users can choose the game they want to run and submit code which can be built and deployed to the cloud system. A resource management service can monitor game console resources between human users and research usage and identify experiments for suspension to ensure enough game consoles for human users.

BACKGROUND OF THE INVENTION 1. Field of the Invention

Embodiments of the invention relates generally managing the use ofelectronic computing resources. More particularly, the invention relatesto methods for managing the use of cloud-based gaming consoles to permitboth research and human gaming use thereof.

2. Description of Prior Art and Related Information

The following background information may present examples of specificaspects of the prior art (e.g., without limitation, approaches, facts,or common wisdom) that, while expected to be helpful to further educatethe reader as to additional aspects of the prior art, is not to beconstrued as limiting the present invention, or any embodiments thereof,to anything stated or implied therein or inferred thereupon.

Video game players often desire to improve their game through practiceand playing against other players. However, once a game player developsexceptional skills in a given game, the availability of suitablechallengers greatly decline. While such players may be able to improvetheir game by playing against less skilled players, it is usually morehelpful to play against a player that can provide a significantchallenge.

Many games provide game-provided players that can participate. However,these players may simply be following certain programming that askillful player can figure out and defeat.

In view of the foregoing, there is a need for a method for training anartificial intelligent agent on cloud-based gaming consoles to have theability to challenge even the best skilled video game players whilemanaging computing resources so that human players are not affected bythe artificial intelligent agent research, development and training.

SUMMARY OF THE INVENTION

Embodiments of the present invention provide a method for managingcomputing resources between human users and research users comprisingproviding a resource management service for the computing resources, theresource management service including: a first resource managementmodule measuring load due to human activity; a second resourcemanagement module predicting future load due to human activity; and athird resource management module determining the computing resourcesgiven to the research users; and providing a resource control servicefor the computing resources, the resource control service including: afirst resource control module reading a number of computing resourcesavailable; and a second resource control module identifying one or moreexperiments to suspend; and moving the one or more experimentsidentified to suspend into a <suspending> state.

Embodiments of the present invention further provide a method fortraining an artificial intelligent agent to play a video game on acloud-based game console shared with human users comprising providingthe artificial intelligent agent to interact in the video game;configuring trainers to review experiences from the artificialintelligent agents and improve policies for the artificial intelligentagents for interacting with the video game; reviewing code for theartificial intelligent agents, the trainers and an experiment definitionprogram with a local source code repository and creating a docker imagethereof; mirroring the local source code control service with a consolesystem source code control service within a game console system buildenvironment and building a docker image for an experiment; monitoring astate of the experiment with a resource control service and determiningwhether to run the experiment once the experiment is in a schedulingstate; starting the experiment on a predetermined number of thecloud-based game consoles, with a predetermined number of the datagatherers, at one or more predetermined environments; providing aresource management service for monitoring an amount of the cloud-basedgame consoles available at each of the environments, the resourcemanagement service including: a first resource management modulemeasuring load due to activity of the human users on the cloud-basedgame consoles at each of the environments; a second resource managementmodule predicting future load due to activity of the human users; and athird resource management module determining the cloud-based gameconsoles given to the research users; reading, by the resource controlsystem, a number of the cloud-based game consoles available;identifying, by the resource control system, one or more experiments tosuspend; and moving the one or more experiments identified to suspendinto a <suspending> state.

Embodiments of the present invention also provide a method for managingcomputing resources of cloud-based game consoles between human users andresearch users comprising providing a resource management service forthe computing resources, the resource management service including: afirst resource management module measuring load due to human activity; asecond resource management module predicting future load due to humanactivity; and a third resource management module determining thecomputing resources given to the research users; and providing aresource control service for the computing resources, the resourcecontrol service including: a first resource control module reading anumber of computing resources available; and a second resource controlmodule identifying one or more experiments to suspend; moving the one ormore experiments identified to suspend into a <suspending> state;assigning the resource control service at each environment containingthe cloud-based game consoles, wherein each resource control servicemonitors for the one or more experiments in the <suspending> state;terminating each process under control of the resource control servicefor the one or more experiments in the <suspending> state; changing astatus of each process under control of the resource control service toa <suspended> state when each process under control of the resourcecontrol service is terminated; and changing the one or more experimentsto a <suspended> state when each of the resource control service at eachenvironment containing the computing resources has its respectiveprocesses for the one or more experiments in the <suspended> state.

These and other features, aspects and advantages of the presentinvention will become better understood with reference to the followingdrawings, description and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

Some embodiments of the present invention are illustrated as an exampleand are not limited by the figures of the accompanying drawings, inwhich like references may indicate similar elements.

FIG. 1 illustrates an exemplary system architecture for training agentsusing game consoles according to an embodiment of the present invention;

FIG. 2 illustrates resources used in the system architecture of FIG. 1 ;

FIG. 3 illustrates a schematic representation of a user computing deviceused in the architecture and methods according to exemplary embodimentsof the present invention; and

FIG. 4 illustrates modules provided by a resource management service anda resource control service for controlling resource use in thearchitecture and methods according to exemplary embodiments of thepresent invention.

Unless otherwise indicated illustrations in the figures are notnecessarily drawn to scale.

The invention and its various embodiments can now be better understoodby turning to the following detailed description wherein illustratedembodiments are described. It is to be expressly understood that theillustrated embodiments are set forth as examples and not by way oflimitations on the invention as ultimately defined in the claims.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS AND BEST MODE OFINVENTION

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the term “and/or” includes any and all combinations of oneor more of the associated listed items. As used herein, the singularforms “a,” “an,” and “the” are intended to include the plural forms aswell as the singular forms, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” when used in this specification, specify thepresence of stated features, steps, operations, elements, and/orcomponents, but do not preclude the presence or addition of one or moreother features, steps, operations, elements, components, and/or groupsthereof.

Unless otherwise defined, all terms (including technical and scientificterms) used herein have the same meaning as commonly understood by onehaving ordinary skill in the art to which this invention belongs. Itwill be further understood that terms, such as those defined in commonlyused dictionaries, should be interpreted as having a meaning that isconsistent with their meaning in the context of the relevant art and thepresent disclosure and will not be interpreted in an idealized or overlyformal sense unless expressly so defined herein.

In describing the invention, it will be understood that a number oftechniques and steps are disclosed. Each of these has individual benefitand each can also be used in conjunction with one or more, or in somecases all, of the other disclosed techniques. Accordingly, for the sakeof clarity, this description will refrain from repeating every possiblecombination of the individual steps in an unnecessary fashion.Nevertheless, the specification and claims should be read with theunderstanding that such combinations are entirely within the scope ofthe invention and the claims.

In the following description, for purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of the present invention. It will be evident, however, toone skilled in the art that the present invention may be practicedwithout these specific details.

The present disclosure is to be considered as an exemplification of theinvention and is not intended to limit the invention to the specificembodiments illustrated by the figures or description below.

Devices or system modules that are in at least general communicationwith each other need not be in continuous communication with each other,unless expressly specified otherwise. In addition, devices or systemmodules that are in at least general communication with each other maycommunicate directly or indirectly through one or more intermediaries.

A description of an embodiment with several components in communicationwith each other does not imply that all such components are required. Onthe contrary, a variety of optional components are described toillustrate the wide variety of possible embodiments of the presentinvention.

A “computer” or “computing device” may refer to one or more apparatusand/or one or more systems that are capable of accepting a structuredinput, processing the structured input according to prescribed rules,and producing results of the processing as output. Examples of acomputer or computing device may include: a computer; a stationaryand/or portable computer; a computer having a single processor, multipleprocessors, or multi-core processors, which may operate in paralleland/or not in parallel; a supercomputer; a mainframe; a supermini-computer; a mini-computer; a workstation; a micro-computer; aserver; a client; an interactive television; a web appliance; atelecommunications device with internet access; a hybrid combination ofa computer and an interactive television; a portable computer; a tabletpersonal computer (PC); a personal digital assistant (PDA); a portabletelephone; application-specific hardware to emulate a computer and/orsoftware, such as, for example, a digital signal processor (DSP), afield programmable gate array (FPGA), an application specific integratedcircuit (ASIC), an application specific instruction-set processor(ASIP), a chip, chips, a system on a chip, or a chip set; a dataacquisition device; an optical computer; a quantum computer; abiological computer; and generally, an apparatus that may accept data,process data according to one or more stored software programs, generateresults, and typically include input, output, storage, arithmetic,logic, and control units.

“Software” or “application” may refer to prescribed rules to operate acomputer. Examples of software or applications may include code segmentsin one or more computer-readable languages; graphical and or/textualinstructions; applets; pre-compiled code; interpreted code; compiledcode; and computer programs.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks.

Further, although process steps, method steps, algorithms or the likemay be described in a sequential order, such processes, methods andalgorithms may be configured to work in alternate orders. In otherwords, any sequence or order of steps that may be described does notnecessarily indicate a requirement that the steps be performed in thatorder. The steps of processes described herein may be performed in anyorder practical. Further, some steps may be performed simultaneously.

It will be readily apparent that the various methods and algorithmsdescribed herein may be implemented by, e.g., appropriately programmedgeneral purpose computers and computing devices. Typically, a processor(e.g., a microprocessor) will receive instructions from a memory or likedevice, and execute those instructions, thereby performing a processdefined by those instructions. Further, programs that implement suchmethods and algorithms may be stored and transmitted using a variety ofknown media.

The term “computer-readable medium” as used herein refers to any mediumthat participates in providing data (e.g., instructions) which may beread by a computer, a processor or a like device. Such a medium may takemany forms, including but not limited to, non-volatile media, volatilemedia, and transmission media. Non-volatile media include, for example,optical or magnetic disks and other persistent memory. Volatile mediainclude dynamic random access memory (DRAM), which typically constitutesthe main memory. Transmission media include coaxial cables, copper wireand fiber optics, including the wires that comprise a system bus coupledto the processor. Transmission media may include or convey acousticwaves, light waves and electromagnetic emissions, such as thosegenerated during radio frequency (RF) and infrared (IR) datacommunications. Common forms of computer-readable media include, forexample, a floppy disk, a flexible disk, hard disk, magnetic tape, anyother magnetic medium, a CD-ROM, DVD, any other optical medium, punchcards, paper tape, any other physical medium with patterns of holes, aRAM, a PROM, an EPROM, a FLASHEEPROM, any other memory chip orcartridge, a carrier wave as described hereinafter, or any other mediumfrom which a computer can read.

Various forms of computer readable media may be involved in carryingsequences of instructions to a processor. For example, sequences ofinstruction (i) may be delivered from RAM to a processor, (ii) may becarried over a wireless transmission medium, and/or (iii) may beformatted according to numerous formats, standards or protocols, such asBluetooth, TDMA, CDMA, 3G, 4G, 5G and the like.

Embodiments of the present invention may include apparatuses forperforming the operations disclosed herein. An apparatus may bespecially constructed for the desired purposes, or it may comprise ageneral-purpose device selectively activated or reconfigured by aprogram stored in the device.

Unless specifically stated otherwise, and as may be apparent from thefollowing description and claims, it should be appreciated thatthroughout the specification descriptions utilizing terms such as“processing,” “computing,” “calculating,” “determining,” or the like,refer to the action and/or processes of a computer or computing system,or similar electronic computing device, that manipulate and/or transformdata represented as physical, such as electronic, quantities within thecomputing system's registers and/or memories into other data similarlyrepresented as physical quantities within the computing system'smemories, registers or other such information storage, transmission ordisplay devices.

In a similar manner, the term “processor” may refer to any device orportion of a device that processes electronic data from registers and/ormemory to transform that electronic data into other electronic data thatmay be stored in registers and/or memory or may be communicated to anexternal device so as to cause physical changes or actuation of theexternal device.

The term “agent” or “intelligent agent” or “artificial agent” or“artificial intelligent agent” is meant to refer to any man-made entitythat chooses actions in response to observations. “Agent” may referwithout limitation to a robot, to a simulated robot, to a software agentor “bot”, an adaptive agent, an internet or web bot.

Broadly, embodiments of the present invention provide an artificialintelligent agent can act as a player in a video game, such as a racingvideo game. The game can be completely external to the agent and can runin real time. In this way, the training system is much more like a realworld system. The consoles on which the game runs for training the agentare provided in a cloud computing environment. The agents and thetrainers can run on other computing devices in the cloud, where thesystem can choose the trainers and agent compute based on proximity toconsole, for example. Users can choose the game they want to run andsubmit code which can be built and deployed to the cloud system. Aresource management service can monitor game console resources betweenhuman users and research usage and identify experiments for suspensionto ensure enough game consoles for human users.

Referring to FIGS. 1 through 4 , the basic workflow can be envisioned asfollows.

On a User's Local Machine

On the user's local machine 10, the user can write a computer program(usually in Python, for example) for the agent. This program is called a“data gatherer” 12 and such an agent can be programmed to know how tointeract with and control a game. The user can further write a computerprogram (usually in Python, for example) for the “trainer” 14. Thetrainer 14 can be programmed to know how to take experiences from datagatherers 12 and use them to improve policies for the agent (datagatherer 12). The trainer 14 may use any number of algorithms and neuralnetwork structures as may be present in an artificial intelligence (AI)library 16. The user can write a third program which defines theexperiment 18. This program is typically in the form of a configurationfile, written in, for example, a human-readable data-serializationlanguage, such as YAML, that can define how many data gatherers 12 touse, how much computing power is needed for the data gatherers 12 andtrainers 14, what algorithms the trainer 14 should use, the set of tasksfor the trainer 14 to put the data gatherers 12 through, and the like.

The user can check in their code (data gatherer 12, trainer 14 andexperiment definition 18) to a source code repository, such as GitHub22. The user can run a command line program, via a command lineinterface 23, that submits a request to the build system 26 in buildenvironment 20 to build the experiment if no existing docker image canbe reused. The user then tells server 52, in the monitoring environment57, via data query interface 25, asking it to run the experimentidentified by its source code check-in reference hash. The system server52 can store information about the requested experiment in a database 56with the state <submitted>. In some embodiments, there may also be a webinterface 24 that lets a user request a run. As shown in FIG. 1 , theweb interface 24 and command line interface 23 can interact with a dataquery and manipulation interface 25, such as Hasura/GraphQL, to permitthe user to review experiments during or after their execution, asdiscussed below. Of course, other query interfaces may be utilized forthe review of data by the user.

In the Cloud

In the cloud computing environment, a build system 26 can build theuser's code into a docker image 28. The build system 26 can be anyvirtual machine imaging system, such as CircleCI, for example. If theexperiment requires resources from the cloud game system 30 (alsoreferred to as production build environment 30), the production buildenvironment 30 can pull code from the development build environment 20,where their build system can run a variety of secondary securityevaluations with a source code repository 32, such as GitLab, and thenalso build the user's code with a docker build 34 into a docker image36. The system can set the experiment state to <building> and recordwhich environment (such as data center DC-1 (environment 38) and datacenter DC-2 (environment 40), as shown in FIG. 2 ) are building it.While the description herein describes using a docker image and FIG. 1illustrates Kubernetes 44 as a container orchestration system forinterfacing with the docker image 36, it should be understood that othertypes of architecture may be used to obtain the same purpose. Forexample, the docker runtime may be replaced by a runtime that iscompliant with the container runtime interface of Kubernetes. Similarly,container orchestrations system Kubernetes can be replaced with otherorchestration systems like Slurm.

Periodically, the resource control service 42 in each environment 38, 40can look at the build system 30 in its view and look for buildingexperiments. When one completes, the resource control service 42 cantransition the experiment state in its environment to a <built> state.The system can watch for the transitions to <built> in each environmentand once all required environments are done, can change the overallexperiment state to <built>. The system can watch for experiments in the<built> stage and transfers them to a <queued> state.

Periodically, the system can evaluate the experiments in the <queued>state and can make decisions about whether an experiment should bestarted. When deciding whether an experiment should start, the systemcan consider the priority level of the experiment, the age of theexperiment, whether the resources the experiment requires are availablein any acceptable environment, and other such criteria for schedulingthe experiment, such as quota limits by user or project, and the like.

If the system decides to start an experiment, it can mark the experimentas <scheduling> and can tag the experiment with identifiers for theresources it should consume. For example, the system may decide that aparticular experiment should be run with game consoles 46 (such asPS4's, for example) and with data gatherers 12 in a particularenvironment 38, 40. The experiment can be run using a GPU (such as V100GPUs 48) for the trainers 14 in the same or different environment andwill add annotations to the experiment to record those decisions.

Periodically, the resource control service 42 in each target environmentcan look at whether there are experiments in the <scheduling>state thatare tagged to start in its environment 38, 40. If so, it can start therequired resources.

When a Data Gatherer Starts

Technically, a data gatherer can be any program. In the context ofembodiments of the present invention, the data gatherer 12 can be onethat is playing a game (such as a PlayStation® game) within the networkof the cloud game system production environment 50.

The data gatherer 12 can find the trainer 14 it is working with asspecified by the system server 52 and connect to it. The data gatherer12 can request a game system user ID from a service that manages userIDs for training agents. The data gatherer 12 requests an availableconsole 46 in the cloud gaming system 50 and also requests a particulargame be loaded.

The data gatherer 12 can then request a task from the trainer 14. Tasksare essentially configurations of the game that it should play. Forexample, in a racing game, one task might have the data gatherer 12start clusters of five cars spaced evenly around the track in which eachcluster contains one car controlled by the agent and three carscontrolled by the game's built-in AI.

The data gatherer 12 can start the game, communicate the scenarioconfiguration to the game, and then start playing. As the agent playsthe game, it can send its experiences to the trainer 14.

Periodically, the data gatherer 12 can fetch updated models from thetrainer 14. Optionally, the data gatherer 12 may also send metrics tothe database 56 via data query interface 25 during or after thescenario. For example, the data gatherer 12 may report its best laptime. Optionally, the data gatherer 12 may store other data, such ascomplete race data, in a remote data store 58, such as S3. Optionally,the data gatherer 12 may configure the video output of the cloud gameconsole to stream to S3 so it can be viewed later by the experimenter.

When the task termination criteria are met, the data gatherer 12 canterminate the scenario on the cloud game console 46 and can request anew task from the trainer 14.

When a Trainer Starts

The trainer 14 can initialize a buffer where it can store experiencesreported by the data gatherers 12. Optionally, a buffer from a previousrun can be loaded. The trainer 14 can maintain a list of tasks fromwhich it hands out new tasks to data gatherers 12 when they request one.

Periodically, the trainer 14 loads experiences from the buffer and useslearning algorithms to update the neural network models. Optionally, thetrainer 14 will report metrics to the system, where such metrics arestored in the metrics database 56. Updated neural network models can besent to the data gatherer 12.

On the User's Machine

While an experiment is building and running, the user can monitor itusing, for example, a web browser 24 connected to the system server 52via data query interface 25. The system interface can show the progressthrough the experiment building and deployment stages. Once theexperiment is running, the system interface can allow the user toinspect metrics and create dashboards displaying various graphs ofperformance. The system interface can also be used to graph metricsacross multiple runs at the same time to allow users to compare theperformance of different experiments.

Suspend and Resume

The resource management service 60, also referred to simply as resourcemanager 60, is the name of the service that the cloud game system has tocoordinate resources with external services. Because the training isperformed on actual game consoles, the training system shares the gamesystem (such as the PlayStation® network) with humans. When more humanswant to play games, resource management service 60 tells the trainingsystem 50 to scale back usage. When humans stop playing, resourcemanagement service 60 gives the training system 50 more resources.System server 52 makes use of resource control service 42, also referredto as experiment manager 42, to make adjustments in resource use basedon targets set by the resource management service 60.

As discussed in greater detail below, some key features of theintegration of the training system with the cloud game system are asfollows: (1) a module 62 that measures load due to human activity; (2) amodule 64 that predicts future load; and (3) a module 66 that determineshow many of those resources can be given to researchers. The resourcecontrol service 42 can provide the following features, including (4) amodule 68 that reads the number of resources available; (5) a module 70that starts and stops experiments according to the resource constraintsand the priorities/age/quotas of the job; and (6) a module 72 thatrestarts jobs in environments where resources are available. Modules 70and 72 may be part of the system server 52. In some embodiments, anexperiment can be run in multiple environments, while the resourcecontrol service 42 (the experiment manager 42) only controls resourcesin one environment. In some embodiments, for example, if the systemserver 52 does not act, or does not act quickly enough, the resourcemanagement service 60 may end experiments according to a pre-programmedprotocol, such as first-in, first-out, for example.

The training system can monitor the cloud game system's resourcemanagement service 60. When the system notices that the resources(especially cloud game consoles) allocated to the training system havedecreased below the system current usage, the system can identify one ormore experiments to suspend. When deciding which experiments to suspend,the system may consider location of the resources in use by theexperiment, priority level of the experiment, age of the experiment,user ID of the experiment and/or other attributes on the experiment. Thesystem can move the selected experiment into a <suspending> state.

Each resource control service 42 in each environment (such as locations38, 40) can periodically check the system server to see if an experimentthey are running has moved into a <suspending> state. If so, theresource control service can terminate the processes under theircontrol. When a trainer is asked to suspend, it can save stateinformation (particularly its experience buffer) to remote storage sothat it can be reloaded later before gracefully shutting down.

Once all of the processes under their control are terminated, theresource control service 42 will change their portion of the experimentto a <suspended> state.

When all of the relevant resource control services 42 have transitionedtheir portions to <suspended>, the system can transition the wholeexperiment to a <suspended> state.

When the system sees in the resource control service 42 that the numberof available resources is greater than the number of resources in use,the system can look at the list of runs that are suspended. The systemmay restart some of these experiments.

The choice about which experiments to restart may consider location ofthe resources in use by the experiment, priority level of theexperiment, age of the experiment, user ID of the experiment and/orother attributes on the experiment.

To avoid thrashing, the system server may smooth the signals aboutresource availability that it receives from the resource managementservice 60. It may smooth these signals by applying any number ofstandard algorithms, like low-pass filters, min-max time windows, or thelike. Optionally, the user can click a button to suspend an experimentthat is running. This experiment will move to the <manually suspended>state. The user may choose to reactivate a manually suspended experimentby pressing a button in the user interface. The system will move theexperiment to <suspended>.

The system will then reactivate the experiment when resources areavailable, subject to the same conditions as above.

Completion

A user may write termination conditions into their trainer script sothat when certain conditions are met, it will report that it completedto the system and then terminate. The system will change the experimentstate to <success>. Alternatively, the user may use the system'sinterface to click the “Cancel” button. The system will shut down theexperiment immediately following a process similar to the suspendprocess discussed above, but without saving the current experimentstate. The system will set the experiment state to <canceled>.

Many alterations and modifications may be made by those having ordinaryskill in the art without departing from the spirit and scope of theinvention. Therefore, it must be understood that the illustratedembodiments have been set forth only for the purposes of examples andthat they should not be taken as limiting the invention as defined bythe following claims. For example, notwithstanding the fact that theelements of a claim are set forth below in a certain combination, itmust be expressly understood that the invention includes othercombinations of fewer, more or different ones of the disclosed elements.

The words used in this specification to describe the invention and itsvarious embodiments are to be understood not only in the sense of theircommonly defined meanings, but to include by special definition in thisspecification the generic structure, material or acts of which theyrepresent a single species.

The definitions of the words or elements of the following claims are,therefore, defined in this specification to not only include thecombination of elements which are literally set forth. In this sense itis therefore contemplated that an equivalent substitution of two or moreelements may be made for any one of the elements in the claims below orthat a single element may be substituted for two or more elements in aclaim. Although elements may be described above as acting in certaincombinations and even initially claimed as such, it is to be expresslyunderstood that one or more elements from a claimed combination can insome cases be excised from the combination and that the claimedcombination may be directed to a subcombination or variation of asubcombination.

Insubstantial changes from the claimed subject matter as viewed by aperson with ordinary skill in the art, now known or later devised, areexpressly contemplated as being equivalently within the scope of theclaims. Therefore, obvious substitutions now or later known to one withordinary skill in the art are defined to be within the scope of thedefined elements.

The claims are thus to be understood to include what is specificallyillustrated and described above, what is conceptually equivalent, whatcan be obviously substituted and also what incorporates the essentialidea of the invention.

The invention claimed is:
 1. A method for managing computing resourcesbetween human users and research users to ensure a sufficient amount ofthe computing resources for the human users, the method comprising:providing a resource management service for the computing resources ineach of one or more could-based environments, the resource managementservice including: a first resource management module measuring load dueto human activity in each of the one or more cloud-based environments; asecond resource management module predicting future load due to humanactivity in each of the one or more cloud-based environments; and athird resource management module determining the computing resourcesgiven to the research users in each of the one or more cloud-basedenvironments; and providing a resource control service for the computingresources in each of the one or more cloud-based environments, theresource control service including: a first resource control modulereading a number of computing resources available; and a second resourcecontrol module identifying one or more experiments to suspend; movingthe one or more experiments identified to suspend into a <suspending>state; terminating, by the resource control service in each of the oneor more cloud-based environments, processes under control of theresource control service; and saving state information of a trainerrunning the terminated processes to a remote storage.
 2. The method ofclaim 1, wherein the computing resources are cloud-based game consoles.3. The method of claim 1, further comprising determining the one or moreexperiments to suspend based on at least one of a location of thecomputing resources in use by the one or more experiments, a prioritylevel of the one or more experiments, an age of the one or moreexperiments, and/or a user ID of the one or more experiments.
 4. Themethod of claim 1, further comprising assigning the resource controlservice at each environment containing the computing resources, whereineach resource control service monitors for the one or more experimentsin the <suspending> state.
 5. The method of claim 1, further comprisingchanging a status of each process under control of the resource controlservice to a <suspended> state when each process under control of theresource control service is terminated.
 6. The method of claim 5,further comprising changing the one or more experiments to a <suspended>state when each of the resource control service at each environmentcontaining the computing resources has its respective processes for theone or more experiments in the <suspended> state.
 7. The method of claim1, further comprising restarting a suspended experiment when thecomputing resources are determined to be available by the resourcemanagement service.
 8. The method of claim 7, further comprisingselecting the suspended experiment to restart based on at least one of alocation of the computing resources in use by the one or moreexperiments, a priority level of the one or more experiments, an age ofthe one or more experiments, and/or a user ID of the one or moreexperiments.
 9. The method of claim 7, further comprising smoothing asignal regarding an availability of computing resources to avoidthrashing.
 10. The method of claim 1, further comprising providing auser with an option to manually suspend and manually resume anexperiment.
 11. A method for training an artificial intelligent agent toplay a video game on a cloud-based game console shared with human users,comprising: providing the artificial intelligent agent to interact inthe video game; configuring trainers to review experiences from theartificial intelligent agents and improve policies for the artificialintelligent agents for interacting with the video game; monitoring astate of the experiment with a resource control service and determiningwhether to run the experiment once the experiment is in a schedulingstate; starting the experiment on a predetermined number of thecloud-based game consoles, with a predetermined number of the datagatherers, at one or more predetermined environments; providing aresource management service for monitoring an amount of the cloud-basedgame consoles available at each of the environments, the resourcemanagement service including: a first resource management modulemeasuring load due to activity of the human users on the cloud-basedgame consoles at each of the environments; a second resource managementmodule predicting future load due to activity of the human users; and athird resource management module determining the cloud-based gameconsoles given to the research users; reading, by the resource controlsystem. a number of the cloud-based game consoles available;identifying, by the resource control system, one or more experiments tosuspend; and moving the one or more experiments identified to suspendinto a <suspending> state.
 12. The method of claim 11, furthercomprising determining the one or more experiments to suspend based onat least one of a location of the cloud-based game console in use by theone or more experiments, a priority level of the one or moreexperiments, an age of the one or more experiments, and/or a user ID ofthe one or more experiments.
 13. The method of claim 11, furthercomprising assigning the resource control service at each environmentcontaining the cloud-based game console, wherein each resource controlservice monitors for the one or more experiments in the <suspending>state.
 14. The method of claim 13, further comprising: terminating eachprocess under control of the resource control service for the one ormore experiments in the <suspending> state; changing a status of eachprocess under control of the resource control service to a <suspended>state when each process under control of the resource control service isterminated; and changing the one or more experiments to a <suspended>state when each of the resource control service at each environmentcontaining the computing resources has its respective processes for theone or more experiments in the <suspended> state.
 15. The method ofclaim 14, further comprising saving state information into a remotestorage before terminating each process.
 16. The method of claim 14,further comprising restarting a suspended experiment when the computingresources are determined to be available by the resource managementservice.
 17. A method for managing computing resources of cloud-basedgame consoles between human users and research users to ensure asufficient amount of cloud-based game consoles for the human users, themethod comprising: providing a resource management service for thecomputing resources, the resource management service including: a firstresource management module measuring load due to human activity; asecond resource management module predicting future load due to humanactivity; and a third resource management module determining thecomputing resources given to the research users; and providing aresource control service for the computing resources, the resourcecontrol service including: a first resource control module reading anumber of computing resources available; and a second resource controlmodule identifying one or more experiments to suspend; moving the one ormore experiments identified to suspend into a <suspending> state;assigning the resource control service at each environment containingthe cloud-based game consoles, wherein each resource control servicemonitors for the one or more experiments in the <suspending> state;terminating each process under control of the resource control servicefor the one or more experiments in the <suspending> state; saving stateinformation of a trainer running the terminated processes to a remotestorage; changing a status of each process under control of the resourcecontrol service to a <suspended> state when each process under controlof the resource control service is terminated; and changing the one ormore experiments to a <suspended> state when each of the resourcecontrol service at each environment containing the computing resourceshas its respective processes for the one or more experiments in the<suspended> state.
 18. The method of claim 17, further comprisingdetermining the one or more experiments to suspend based on at least oneof a location of the cloud-based game console in use by the one or moreexperiments, a priority level of the one or more experiments, an age ofthe one or more experiments, and/or a user ID of the one or moreexperiments.